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ABSTRACT 

An interactive test analysis system wets developed - 
which interfaces a 3M DATEOHICS system with a XEROX. Sigma 9 computer* 
The computer progijams were written in A Programming Language (APL) « 
The current .impleriientation of the program is characterized ,by its 
capability to: read responses from a DATROJIIC answer sheet;^ allow the 
faculty member the option of omitting ite^s jErom the analysis; check 
for omitted items; allow the faculty member the* option of selecting 
specific reports; and collect descriptive statistics concerning the . 
system performance and characteristics of the test analyzed. The 
complete set of reports generated by the. test analysis program 
provide information normally included in an analysis of a 
norm-referenced test. The fixed costs for the test analysis system 
are the 3h Datronic System and the Anderson Jacobson terminals If you 
assume a lease option for each component of the system then the 
approximate cost will be $240, per month.. This fixed cost can be 
allocated on the assumption of full utilization of the system with an 
average cost of $1.50 per hour for a 40 hour w^ek. Hhen the costing 
formula was applied to the data base of 88 sets of testsr the average 
and median cost were determined to be $8.37 and $7.83 respectively. 
(Author/RC) 
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Interactive Test Analysis^ • ' 
Development, Imp 1 imen t a t i on » and ^ Eva 1 u at i on ^' 

The purpose of this paper is to describe the develop me nt, 
implementation, and evaluation of an interactive test analysis 
system* The primary focus of the paper will be on the operational 
characteristics of'the system and the problems associated with 
determining the costs of such services in an educational setting. 

DEVELOPMENT 

The TJZ .U , Int erac t ive Test Analysis Program was^ deve loped to 
provide test scoring and test analysis services to the member^ of 9 
the T.C.U. instructional staff. The initial performance standards 
established for the test analysis services, 93% completed within one 
working day and 1002 completed within two working days, has been 
achieved by the system* 

The three phases of any data analysis prograrat' data etitry^ 
data analysis, and report generation are accomplished through the 
interface of the: 

A. 3M DATRONICS Data Analysis System, 

B. Anderson Jacobson Selectronic 841 Terminal, and 

C . XEROX Sigma9computer. ' 

Each of ^ the thr6e components of tj^ie system has proved highly 
reliable and the versatility of the system, in an educational 



eilvironraent, appears to be vi-ry prcat^ Information concernitig the 
^ components of ttie system can he obtained' from the manufacturer. (See 
• Appendix 1 ) ' ' ^ 

The XEROX Sigma 9^ computer syst;em provides the 04>^i6n of 
selecting FORTRAN, BASIC, or APL as the programming language for the 
test analysis system. A Programming Language (APL) was seljected on 
the basis of its ability to manipulate and perform computations 
involving matrices of numeric data. APL,' as it has been itnp^J^e^en t ed 
on the XEROX Sigma 9 computer (XEROX APL - Sigma 6/7/9 Computers: 
Language -and Operations Reference Manual) enables the user to copy 
functions from a library and tb erase variables or functions from 
the active workspace during execution of a function. This feature 
; greatly reduces the size of the -workspace required to 'analyze *the 

testdata. . *. ^ 

The author served as both the designer and programmer for the 
Zjl£jJI-1 iili££2£iiH Test Analys is Pr ogram . The programs were 
developed, tested, and modified over a period of t^hree months. 



Accurate e-stimates of the time required to develop the program's 
would hav9 beep possible if the author had filed the relports 
generated by the XEROX operating system. Existing APL functions, 
from the T.C.U. Computer Center library were utilized wljen it was 
feasible to do so* The major portion "of the programs we're developed 
in an interactive mode. Each APL function was programed, tested; 
modified, and integrated into the main control function named START 
(See Appendix 2). Faculty Members, who h^d previously uti 1 ize'd| the 
test scoring service, were consulted both as to the. informati6n they 
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wanted rece-iv-e and the format developed for the* final reports. 
Requests for additional information to be included in the reports 
and the need to verify the accuracy" of the data entered, often 
required major program mod i fj[,cat ion . 

Three factors which make the set of APL programs longer and 
less efficient than typical APL programs are the: 
^ 'a. use of comments to internally document the functions, 
' B. inculsion of the interpretative information in each 
report generated by the system, and 
C. ability of the propr amra.er to maximize the power 
and efficiency of the APL language. 

* » • 

IMP-LEMENTATION 

The test analysis programs have been available for only three 
months, March through May 1975, and during this period of time 
approximately 100 sets of tests have been analyzed by Instructional 
Services. Sets of tests were eliminated form the data base if the 
number of ^'tudents or items was less than ten of if the pe^^formance 
data was known to be invalid. The invalid data occurred most 
frequently in the early stages of development when unexpected 
pr.oblems would be discovered in the ope rat ion ' of the system. The 
resulting data base consisted of complete data for 88 sets of tests 
papers. 

The current implementation of the program .is characterized by 

itscapabilityto: f% 

V ^ : . 

A. read responses from a DATRONIC answer sheet. 
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•selecting specific reports, and - • 

F. collect descriptive statistics concerning the 
system performance and characteristics of the 
test analyzed • ^ • 
The complete- set of reports 'generated by the test analysis 
program (See Appendix 3) provide the information normally included 
in an analysis of a^norra refererfced test, such as thef 

A, descriptive statist ics for the test, 

B. table of frequeacy and ^cummulative frequency, 
€• histogran of grouped data^ 

D. table of raw and converted scores, 

£• item difficulty and discrimination indexes, and 

F. reports (ff individual studen^ performance. 

Program documentation is cur rent lX be ing prepared and the ^ 
J- ' \ ^ 

entire set of programs will oe sabmLt ted \o the XEROX Users" group. 

The use of imbedded commands tcJ copy or erise functions or variajjle 

and the deltd p-MT function are the '0017 n^n standard APL features- 

us£d in the* programs. - — ^ ■ 



; ' EVALUATION 

, The test analysis systen. has proved to be highly reliable and 
^has met the performance specification's, 952 completed in 1 working 
day and lOOX cqmp.leted within 2 working days, established by 
Instructional Services. The objective, in all educational services, 
is to provide the services as cost effectively as possible. In the 
analysis of costs, fixed and variable costs will be discussed 
separately • 

The fixed costs for the test analysis system are the 3M 
Datronic System and the Anderson Jacobson terminal. The optionsof: 
purchase, lease - purchase, nnd rental are available for each 
component of the system. T.C.U. has purchased the 3M Datronics 
system ($3,300) and leased the Anderson Jacobson terminal. If you 
assume a lease option for each component of of the system then the 
approximate costs will be: 

3M Datronic System - $118 per month (3- yr^J 

Artsderson Jacobson terminal - $122 per month 

Total - $240 per month 
f 

These fixed cost can be allocated on the assumption of full 
utilization of the system with and average cost of, $1.50 per hour 
for a 40 hour week or $.75 per hour for an ^0 hour week. It is 
unlikely that institutions with student bodies of less than 10,000 
students need to allocate more than 40 hours per week for test 
analysis services. In subsequent calculation, $1.50 per hour is 
used for the'fixed costs. 
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The variable costs involved in tne Cest analysis system are: 

A. Student Operator • $ .03 minute 

B* Terminal Connect Time- .05 

C* £PU time - 3.25 

D» Answer Sheets • ^025 

£. Paper Cost ♦ - .10 per set. 

The formula used to determine the estimated cost of eaqh test 
analyzed by Instructional Services is: 

COST = Connect time x ^025 

+ Connect time.x ,U3. ' 

+ Connect time x .05 ^ ' 

+ CPU time X 3.25 ' 
+ Answer Sheet (§ .025 
+ Paper @ . 10 . 

*When the formqla was applied to the^total data base, 88 aets .of 

\ 

tests, the average and median cost -^were determined to be §8.37 and | 
$7.83 respec^iively. As expected, the average and median costs 
increased as the lenpht of tne test in-creased: 

AVERAGE MEDIAN 
10 • 50 item test * - = $6.64 $ 6.66 

51 • 101 item test = 10.11 ' ' 10.85 ' ^ ' 

101 - 150 item test = 15.69 17.29- 

Although the cost of the service is substantial, it is less than the 
same information could be derived through hand^ ca l.cu lat ion of the 
sametestanalysis information. » ^ 
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DISCUSSION 

The evalua t ion^of the ^T^^U^Ln tercet iv£_Te9£^ Program 
points out the complexity of developing such" a system and the need 
to develop techniques for reducing the cost of the services 
provided., Some of the chances which^'would contribute to delivering 
comparable test analysis services at reduced cost include: 
'A. Delete tne descriptive in f orma t i on . f r om the 
APL programs and prepare a user'^s guide, 

B. Development of automatic feed device for the 
test scoring machine (this is currently under 
development by 3M Corporation), 

C. .Development of test scoring machine which 

would read both sides (lUO iteriLs) at one pass, 

D. Remote printing of APL generated files, and 
£. Increase the efficiency of the APL programs.. 

It has been estimated that implementation of the first , * 
suggestion will reduce the a.verage cost of the set of tests analyzed 
from $8,37 to $6.67. The remaining improvements require the 
continued cooperation between the manufacturers of the equipment and 
the developets of test analysis programs and systems. 

The developer of test analysis programs must be in constant 

ii 

communication with the faculty members who utilize the program in 
order to assure that the information provided is being accurately 
interpreted -and that the information needs of the user are be met by 
the program. Some of the major program developments which will 



enhance the current test analysis programs include: 

A, capability to analyze and report subtest scores, 

B, inf ormat ion managemen t system to store and 
and accumulate student and 'item statistics, 

C, computer generated tests, and 

♦D, test analysis programs for criterion referenced tests. 
The test analysis system has also demonstrated that it can 
support faculty evaluation and during the summer of 1975 it will be 
used by the student government in ana 1 y z i ng ' the i r election results. 
Analysis of the cost of usini? the system to read the 25 ,000 student 
evaluations into data files is currently under study. 



/ » 
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SYSTEM COMPONENTS: SOURCES AND COST ESTIMATES 



3M DATRONICS Data Analysis S^ystem 
5500-RST Data Analysis Svstem 



Mr. Jim Womac 
3M Center 
3M Corporation 
Building 209-1 
St. Paul, Minn. 
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Anderson-Jacobson Computer Terminal 
AJ SELECTRONIC 841 



Mr. George Fish 
10 6.5 Morse Avenue 
Sunnyvale, CA 94086 



XEROX Computer Users'* Group 

7U1 South Aviation Boulevard 
El Segundo, CA 90245 * 
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